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ABSTRACT 



The iinportance of experiments in educational research is widely recognized, 
but methods used to analyze experimental data may not always be fully appropriate. 
Advantages and disadvantages of three analytic methods are discussed. A 
hypothetical data set is enployed to make the discussion concrete. It is 
suggested that oomnonality analysis can be usefully employed in research studies 
in education, particularly when aptitude-treatnnent interaction studies are. 
involved. 



Researchers have long recognized the irnportance of experimental designs in 
^ social science. As WUndt observed in 1904, "The experimental method is of 
cardinal inportance; it and it alone makes a scientific introspection possible" 
(p. 4). However, ^ Winer (1962, p. 2) noted, the design of an experiment can 
make or break a study: 

Two experiments having identical objectives may be 
designed in quite different ways; at the same cost l.i 
terms of experimental effort, one design may lead to 
unambiguous results no matter what the outcome, whereas 
the second design oould potentially lead to ambiguous 
results no matter what the outcome. 
It was this recognition that some 60 years agoXprompted McCall (1923) to publish 
his text. How to Experiment in Education . Continuing concerns regarding the 
inportance of designs have stimulated nore recent works as well (Campbell & 
Stjanley, 1966; Cook & Cairpbell, 1971) . 

Concerns, regarding the analysis of experimental data also have their own long 
history. In 1925, in a work titled Statistical Methods for Research Wbrkers , Sir 
Ronald Fisher presented the analysis of variance (TtfJOVA) techniques that he had 
developed. Today ANOVA methods and their analogs (ANCa'A, MANOVA, and 
MANOOVA— collectively here labelled OVA met-hods) are among the most comnonly 
applied methods in the social sciences (Edgington, 1974? Wick & Dirkes, 1973? 
Willson, 1980). 
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Unfortunately, although design and analysis were initially conceptualized as 
discrete issues, it has now becane quite oomnon to confuse design with analysis. 
Specifically, om, techniques have become somewhat equated with exfierimental 
design, and vice versa. Hicks (1973) provides a classic example of this coiifusion 
in his fine book. Fundamental Concepts in the Design of Experijnents . The book is 
entirely about OVA analysis rather than about design. Thonpson (1981) explores 
the etiology of this confusion. 

The purpose of this paper is to point out that data from experiments can be 
profitably analyzed in a number of related but distinct ways. The advantages and 
limits of both OTA and non-OVA techniques are discussed. One , hypothetical data 
set is analyzed throughout the presentation in order to make the discussion wore 
concrete. Although the data set aiJ the analyses are presented as a one dependent 
variable, or ''univariate" case, it can be shown that the discussion generalizes to 
the multivariate case as well. 

Method One; ANOVA 

Table 1 presents data from a hypothetical experiment involving 24 subjects. 
The data are presented in some detail and using variable names accepted by several 
computer packages in case readers wish to verify or explore certain results using 
their own oomputers. Twelve subjects each were assigned to either. a control 
condition (EXPE3RGRP=1) or an experimental condition (EXPERGRP=2) . The 
hypothetical study also involves a measure of the subjects' aptitudes for 
mastering the dependent variable task. In this case both the aptitude variable 
and the dependent variable were cognitive tasks, i.e., respectively performance on 
an IQ test and standardized (Z) scores on an achievement posttest (ZDV) 
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I .1 
administered at the oonpletion of the experiment. 

INSERT TABLE 1 ABOOT HERE. 

This is an exanple of the classic aptitude-treatment interaction (ATI) design 
that Cronbach (1957, p. 681) touted so convincingly in his 1957 presidential 
address to the American PsycJhological Association: 

Ultimately we should design treatments, not to fit the 
average person, but to fit groups of students with 
particular aptitude patterns. . . fSuch efforts] will carry 
us into an educational psychology which measures readiness 
for different types of teaching and which invents teaching 
methods to fit different types of readiness. 
Notwithstanding both the limits and the advantages of ATI designs (Cronbach, 
4.975), as Kerlinger (1973, p. 257) notes, "in the opinion of some behavioral 
researchers, especially in education, the study of interactions is becoming 
increasingly important and should become a central concern of researchers." 

All independent variables or "ways" must be converted to nominal scale in 
order to perfonr. uvA analyses. In this case the IQ data were converted into a 
trichotatiy (IQGRP) in the usual manner. Table 2 presents conventional ANOVA 
results associated with the 3X2, six cell design. 

INSERT TABLE 2 ABOOT HERE. 

Method Two; Regression Coding 
Cohen (1968) and others recognized quite some time ago that multiple 
regression analysis can readily be enployed to perform OVA analyses. However, the 
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widespread use of regression approaches to OVA analyses has been a more recent 
phenomenon (Willson, 1982) . All that is required is that information about design 
cells be converted into "coding*? variables, as illustrated in Table 1. Although 
discussion of the several methods for acconplishing this conversion is beyond the 
scope of this report, the conversion is clearly presented in several widely 
available texts (Edwards, 1979; JKerlinger & Pedhazur, 1973, pp. 116-153) and in 
practice is a straightforward matter. 

But it should be noted that the coding variables (IQl through EXPBYIQ2) 
represent exactly the same information as presented for the ways in the OVA 
analysis (IQGRP and EXPERGRP) , albeit in a different form. For example, note that 
all subjects in the "low IQ" cells of the IQ way (IQGRP=1) receive coding scores 
of -1-1 for the IQ coding variables, and that subjects in the other cells of the 
way receive different scores (0+2 or +1-1) on the two IQ coding variables. Note 
also that each of the six design cells receive a unique set of coding scores. For 
example, only the high IQ, experimental group subjects receive the coding score 
set: +1-1+1+1-1. 

\ 

The analysis is conducted by entering the five coding column predictor 
variables into a stepwise multiple regression analysis to predict the dependent 
variable, ZDV. The sum of squares (SOS) regression or\ any given step, minus the 
SOS regression on any previous step (if any) , is the SOS for the hypothesis 
associated with the coding variable entered on the given step. The remainder of 
the keyout presented in Table 3 is derived by then manipulating these sum of 
squares using a calculator. 

INSERT TABLE 3 ABOUT HERE. 
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The only difference in the results when oonpared with the ANOVA analysis is 
that tlie coding results partition the total SOS of the dependent variable into 
smaller parts using one degree of freedom hypotheses. But this difference can be 
important whenever any way has more than two levels or cells. The coding analysis 
provides nore specific information about from where the effects of independent 
variables on the, dependent variable arise. 

In the hypothetical example the cjoding column IQl tests the null hypothesis 
that the mean ZDV score of the eight "low IQ" subjects equals the mean ZDV score 
of the eight "high IQ" subjects. The IQ2 coding column tests the null hypothesis 
that the mean ZDV score of the eight "medium IQ" subjects equals the mean ?DV 

» 

score of the remaining 16 subjects. Note in the example that the SOS for the IQ 
way in the Table 2 OVA analysis was .010. The coding analysis breaks this effect 

into two smaller oonponents (IQl SOS = .00000 and IQ2 SOS = .00968) and indicates 

/ 

chat almost all of tb^e effect of IQ (although very minimal) arises from 
differences between the "medium IQ" subjects when conpared with the other 
subjects. 

Of course, more speciric information about where differences occur within an 
OVA analysis can also be determined by conducting post hoc tests after the 
analysis is ccitpleted. Exarrples of OVA post hoc methods include the Scheffe test 
or a multiple range test. However, a priori hypothesis specification via coding 
still may be preferred because, as Kerlinger and Pedhazur (1973, p. 131) note; 
The tests of significance for a priori, or planned, 
coiparisons are more powerful than those for post hoc 
comparisons. In other words, it is possible for a 

o ^ 3 
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specific oonparison to be not significant when tested by 
post hoc methods but significant when tested by a priori 
methods. 

Method Threet Oonmonality Analysis 
Regression coding represents OVA analyses as a regression equation. The use 
of regression coding to analyze data from experimental designs suggests the 
intriguing possibility that multiple regression could be enployed to analyze 
experimental data without having to reduce ordinal or interval independent 
variable data, e.g., aptitude data, to the nominal level of / scale . It is 
recognized (e.g., Nie, Hull, Jenkins, Steinbrenner & Bent, 1975, pp. 372-373) , 
although perhaps not widely, that /interaction can p*:ill be represented in the 
analysis by ocnputing what ^rlinger and Pedhazur (1973, /p. 414) have termed 
"product variables." A product variable' is created by niiltiplying any two 
variables times --each other. Thus, the last three columns of Table 1 present all 
the independent variables necessary to conduct an analysis of experimental data 
without having to convert the IQ data into the nominal level of scale. 

Cohen (1968) has inplied that OVA methods have been attractive to social 
scientist^ because partitioning the dependent ^variable's variance into 
uncorrelajted portions provides "oorputational sinpli^ity" — this may have been 
particularly important in the era preceeding widesprej^d availability of computers. 
However, use of regression with data at the level of scale at which the data are 
originally collected also has advantages. For exanple, as Darlington (1968, 
p. 166) notes: 

In analysis-of-variance designs, the oonplete independence 
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of all the independent variables is assured |>y the 
requirement of equal or proportional cell frequencies. . . 
In nwltiple regression, however, there is no requirement 
that predictor variables be uncorrelated. This property 
gives regression analysis a substantial element of 
flexibility lacking in analysis of variance. 
It has also been observed (Cohen, 1968? Thotpson, 1981) that the use of non-OVA 
nethods does not: , .1) reduce reliability of aptitude and non-experimental 
variables; 2) inflate the Type II error probability? 3) discard substantively 
inportant information? or 4) distort the distribution shapes of or relationships 



among certain variables. 



! 



id 



ERIC 



Although beta weights and structure coefficients (Thcnpson & Borrello, in 
press) can be consulted to evaluate "main effects" and the "interaction effects" 
represer^ted by product variables, cormDnality analysis, also called "element 
analysis" (Newton & Spurrell, 1967) and - "coitponents' analysis" (Mayeske, Wisler, 
Beaton, Vfeinfield, Colien, Okada, Proshek & Tabler, 1969) , can also be conducted if 
the researcher wants estimates of the uniqueness of each effect. For each 
independent variable, commonality analysis indicates how much of the variance of 
the dependent variable is "unique" to the predictor, and how much of the 
predictor's explanatory ori predictive power is "cannon" to or also available from 

/ 

one or more of/ the other predictor variables. Mcxxi (1969) presents an algebraic 
rule tor gomputing these variance partitions for any number of independent 
variables, and Cooley and Lohnes (1976, p. 222) have tabled the required 
coiputational methods for studies involving as many as four independent variables. 
In addition to tabling the ooniputational procedures for studies involving up to 
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five independent variables (pc 358) , SeiboJ.d and WcPhee (1979) provide a brief and 
understandable introduction to this method of analysis., 

In the present study the squared mi;litiple correlation between ZDV and the 
predictor variables, ZIQ, ZEXPGRP, and ZIQBYEXP, was .23708. Table 4 illustrates 
the oonputations of the partitions of the unique and oomnon predictive ability of 
the three independent variables to explain 23.708 percent of the variance in ZDV. 

Table 5 sunmarizes the commonality and regression results. 

i ■ ■ ■ ■ 

INSERT TApLES 4 AND 5 ABOOT HERE. ' 

Two aspects of the oonnDnality analysis merit further explanation. " First, 
the interaction effect represented by the product variable, ZIQBYEXP, should not 
be confused with the oonniDnality involving the two main effect variables, ZIQ and 
ZEXPGRP (Seibold & MciPhee, 1979, p. 365). Interaction is the unique effect of two 
or Kore independent variables which in combination affect the dependent variable. 
Caponality indicates the proportion of predictive- ability of a single variable 
that >lso happens to reside in another single predictor variable too; no unique 
effect of the predictors acting in combination is involved. 

Second, negative carmonalities (never negative uniqueness partitions) can 
occur, as with the "a,b,c'' commonality in the present analysis. This is 
counterintiutive since the result could be taken to mean that each of the three 
predictor variables have in ooramon the ability to explain less than zero percent 
of the. variance in ZDV, , Instead, negative oomnonalities frequently indicate the 
presence of a suppressor effect (DeVito, 1976, p. 12). As Craeger (1971, p. 675) 
notes: 

This [a negative corcnonality variance partition] is nore 

11 
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^ likely to occur in liigher order partitions obtained by 
. subtraction and -is nore likely where some regression 
weights are negative (either by suppressor effects or froin 
bipolar relations which cannot be ^emoved^ by, reflecting 
vectors). Negative partitions may also result from 

? 

I 

. saiipling errors in the correlation matrix. 

• i ' ■ 

Beaton (1973> p. 22) provides an illustration of how a negative ccnironality can 
have important substantive implications: , ' 

Both weight and speed are important to .success as a *^ ' 

• professional football player and each would be inoderately 
correlated with a measure of success in football. Weight 
and speed are presumably negatively correlated and would - 
have a, negative commonality in predicting success in 
football. If both weight and speed are known, one would 
expect to make a much better prediction of success using 
both variables to select fast, heavy men rather than just 
selecting the fastest regardless or weight or heaviest 
regardless of speed. Thus the negative coifHTonality 
indicates that explanatory power of either is greater when, 
the other is used. 

Seibold and McPhee (1979, pp. 364-365) report results of a cancer study that may 
well have been grossly misinterpreted if a commonality analysis detecting 
suppressor effects had not been conducted. 

« 

Discussion 

12 
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The three techniques for analyzing data from experiments each have both 
advantages and disadvantages. For exairple, with respect to significance testing 
for the variance partitions from oannonality analysis, "for .the unique parts, and 
the unique parts only, one can make the usual P test of whether additional 
re^ressicjn terms have contributed ^significantly to the regression if he is willing 
t6 overlook the logical difficulties arising from the fact that the tests are not 
irtdependent" (^'tood,. 1971, "pp. 196-197). However, with respect to the 
09rnnonalities themselves, as Newton and Spurrell (1967V P. 61) note that "it is 
difficult to see that statistical theory will be able to give sampling errors 
which can be used in meaningful tests for secondary elements rcommonalities] since 
thgy are obviously not Independent statistical quantities." 

Still, commonality analysis does inherently focus attention on effects size 
estimates for experimental, aptitude, and other independent variables. This 
emphasis is consistent with the recognition that statistical significance is 
prinarily a function of sanple size (Carver, 1978) , and that estimates such as 
Hays^ (1963, p. 382) omega squared are iitportant adjuncts to OVA analyses. The 
focus is also consistent with enphasis on effect size.s in meta-analysis (Glass, 
McGaw & Smith, 1981) . 

As a practical matter, the advantages of commonality analysis accrue in most 
research examples because so many experiments involve aptitude or other intervally 
scaled independent variables. In fact, most variables other than experimental 
manipulat',on are higher than nominally scaled. Probably the most notable 
exception is sex of the subjects in a study. Although in exceptional areas of 
inquiry such as math anxiety (Aiken, 1976, p. 3^2) the use of sex as an 
independent variable may be warranted, as a general rule the use of even this 
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variable may stem wore from ease of measurement than from theoretical 
justification. \ 

The coirpelling advantage of comnonality analysis of experimental data is that 
the analysis does not require that all independent variables be converted to the 
nominal level of scale. It is this "squandering Fof] much information" (Cohen, 
1968, p. "^441) that causes the previously mentioned difficulties wiUvOVA methods. 
For exanple, it is this feature that distorts relationships among the independent 
variables, and as Seibold and MciPhee (1979, p. 355) argues 

Advancefiient of theory and the useful application of 
research findings depend not 6nly on establishing that a 
relationship exists among predictors and th6 criterion, ' 
but also upon-^ determining the extent to which those 
independent variables, singly and in all possible 
carbinations, isha^re variance with the dependent variable. 
Only then can we fully know the relative importance of 
independent variables with regard |to the dependent 
variable in question. 
Commonality analysis of data from experiments is attractive because the method 
honors the reality to which the researcher is purportedly trying to generalize. 
As Mood (1969, p. 480) notes, "The independent variables in any social process, 
and certainly in education, are highly correlated among thotiselves, and this kind 
of partition of variance [conmonality analysis] provides measures of the extent to 
which they overlap each other in their association with the dependent variable." 
Thus, perhaps commonality analysis of data from experiments should be considered 
more frequently 'in oonteroporary practice in educational research. 
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Table 1 

Exarnple Data Set for Three Related Analyses 



OVA Data C3odinq Data Commonality Data 

Case IQ IQGRP EXPERGRP ZDV IQl IQ2 EXP EXPBYIQl EXPBYIQ2 ZIQ ZEXPGRP ZIQBYEXP 
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+1 


-1 


-1 


-1 


+1 


+1.261 


-0.979 


-1.234 


20 


140 


3 


1 


+1.094 


+1 


-1 


-1 


-1 


+1 


+1.554 


-0.979 


-1.5'z2 


21 


120 


3 


2 


-0.270 


+1 


-1 


+1 


+1 


-1 


+0.716 


+0.979 


+0.701 


22 


132 


3 


2 


+0.412 


+1 


-1 


+1 


+1 


-1 


+0.412 


+0.979 


+1.193 


23 


135 


3 


2 


+0.071 


+1 


-1 


+1 






+1.345 


+0.979 


+1.316 


24 


143 


3 


2 


-1.292 


+1 


-1 


+1 


:k 




•t;1.680 


+0.979 


+1.645 



Note: "ZDV" is the dependent variable for all three analyses. "ZIQBYEXP" is the 
product of "ZIQ" times "ZEXPGRP" (respectively, the Z score versions of "IQ" and 
"EXPERGRP"). ~ 
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Source 
IQC5RP 
EXFERGRP 



Table 2 
I ANOVA Analysis 
Sun of M -an 

Squares df Square Fcalc 
.010 2 .005 <1 

2.135 2.262* 



i 2.135 1 

i 

2 Way Interaction 3.863 2 
Error 16*992 18 

Total. 23.000 23 1.000 

Note : "Effect Size" is an £ squared analog. 



1.932 2.046 
.944 



Effect Size 

.010/23 « .000 
2,135/23 = .093 
3.863/23 = .168 



\ 



Table 3 
Regression C3oding Analysis 





Sum of 




Mean 




Effect 


Source 


Squares 


df 


Square 


Fcalc 


Size 


IQ Group 












Low vs. High (IQl) 


.00000* 


1* 


.00000** 


<1** 


.00000** 


Medium vs. Other (IQ2) .00968* 


1* 


.00968** 


<1** 


.00042** 


EXP 


2.13492* 


1* 


2.13492** 2.26154** 


.09282** 


Interaction 

« 


3.86320 


2 


1.9^160** 


2.04616** 


.16796** 


EXPBYIQl 


3.51464* 


1* 


3.51464** 






EXPBYIQ2 


.34856* 


1* 


.34856** 






Error 


16.99221 


18 


.94401 






Total 


23.00001 


23 


1.0^00 







*By subtraction of one step's results from following step lifted on the printout, 
**By division using previously obtained tabled results. 
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Table 4 

Coninonality Conputations of Variance Partitions 



Unique to ZIQ (a) 



Unique to ZEXPGRP (b) 



2 2 
-R y.bd +R y.abc 
-.23618 +.23708 ~ .00090 

2 2 
-R y.ac +R y.abc 
-.14440 +.23708 = .09268 



Unique to ZIQBYBXP (c) 



Connon to a,b 



Coninon to a,c 



Comnon to b,c 



Coramon to a,b,c 



-R y.ab +R y.abc 
-.09995 +.23708 = .13713 

2 2 2 2 

-R y.c +R y.ac +R y.bc -R y.abc 
-.14335f. 14440 +.23618 -.23708 = .00015 

2 2 2 2 

-R y.b +R y.ab +R y.bc -R y.abc 
-.09282+. 09995 +.23618 -.23708 = .00623 

2 2 2 2 

-R y.a +R y.ab +R y.ac -R y.abc 
-.00677+. 09995 +.14440 -.23708 = .00050 

2 2 2 2 2 2 2 

+R y.a +R y.b +R y.c -R y.ab -R y.ac -R y.bc +R y.abc 
+.00677+.09282+.14335-. 09995 -.14440 -.23618 +.23708 =-.00051 



Note; The sum of the unique and common predictive abilities of the three independent 
variables, represented by the seven partitions, equals the squared multiple correlation 
(.23708) obtained when the three independent variables are used to predict the 
dependent variable, ZDV. 



/ 

ERIC 
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Table 5 

Camonality and Regression Results 
Unique to ZIQ (a) .090% 
Unique to ZEXPGRP (b) 
Unique to ZIQBYEXP (c) 
Cannon to a,b 
CJomnon to a,c 
Common to b,c 
Camon to a,b,c 



9.268% 

13.713% 

.015% 

.623% 
.050% .050% 
.051% -.051% -.051% 
.677% 9.282% 14.335% 



.015% 
.623% 



Sun of the Partitions 
2 

r of predictor with ZD7 .00677 .09'282 .14335 

Beta weight .03151 .30444 -.38804 

Structure Coefficient -.16896 .62572 -.77760 
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